<html>
<head><meta charset="utf-8"><title>Write huge amounts of data to file · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html">Write huge amounts of data to file</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="242560660"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242560660" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Oliver E. Anderson <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242560660">(Jun 14 2021 at 07:23)</a>:</h4>
<p>I want to write millions of lines to a single file. What is the best way to do this keeping performance and safety in mind? <br>
For more context I don't have enough memory to keep everything I want to write to file in memory at once, so maybe <a href="https://doc.rust-lang.org/std/io/trait.Write.html#method.write_all">https://doc.rust-lang.org/std/io/trait.Write.html#method.write_all</a> won't do the job?</p>



<a name="242560780"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242560780" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242560780">(Jun 14 2021 at 07:24)</a>:</h4>
<p>Just layer a bunch of writer adapters over a BufferedWriter and make sure to write everything with an iterator like interface so that you can stream data</p>



<a name="242560878"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242560878" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242560878">(Jun 14 2021 at 07:26)</a>:</h4>
<p><code>write_all</code> is fine, it just means "write all of the provided buffer to the output stream"; the buffer itself can be just a chunk of your desired output. You should just try to avoid putting all data in the buffer before calling <code>write</code>/<code>write_all</code></p>



<a name="242560955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242560955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242560955">(Jun 14 2021 at 07:27)</a>:</h4>
<p>For line-oriented output, it probably suffices to buffer each line and then write it, and then call that a million times in a loop</p>



<a name="242562025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242562025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Oliver E. Anderson <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242562025">(Jun 14 2021 at 07:40)</a>:</h4>
<p>Thank you <span class="user-mention" data-user-id="271719">@Mario Carneiro</span>  :)</p>



<a name="242626932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242626932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> XAMPPRocky <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242626932">(Jun 14 2021 at 16:49)</a>:</h4>
<p><span class="user-mention" data-user-id="390743">@Oliver E. Anderson</span> Performance is decided by the type not the trait for the most part, so you don’t want to look at <code>Write</code>, you want to compare different type’s <code>Write</code> implementations.</p>
<p>In general with large files, you might have better performance with memory mapping the file rather than allocating a normal File. So you want to compare those two.</p>



<a name="242660032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Write%20huge%20amounts%20of%20data%20to%20file/near/242660032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Write.20huge.20amounts.20of.20data.20to.20file.html#242660032">(Jun 14 2021 at 21:17)</a>:</h4>
<p>Really depends on how much you want to optimize, just making sure to avoid small write syscalls while also avoiding memory pressure can get you pretty far, either by reusing reasonably large buffers directly or using a <code>BufWriter</code> to coalesce writes. Memory mapped IO tends to be quite a mixed bag, the page faults can be more expensive than writes with large buffers, you definitely have to measure that.</p>
<p>Optimizations beyond that point take more effort, e.g. doing the processing on multiple threads and using absolute positioned writes or using async IO APIs (very OS-specific). Separating the compute work from the IO can help too.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>